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EXECUTIVE  SUMMARY 


Battel le's  Columbus  Laboratories  (BCL)  has  completed  raster- to- vector 
benchmark  testing  on  the  Scltex  and  Broomall  cartographic  data  capture 
systems  at  the  Defense  Mapping  Agency  Hydrographic/Topographic  Center 
(DMAHTC).  A  cartographic  benchmark  testing  package  and  testing 
methodology,  developed  during  the  "DMA  Raster-to-Vector  Analysis"  project, 
were  validated  during  the  testing  on  DMA  production  systems.  This  final 
report  consists  of  six  topical  areas:  1)  Benchmark  testing  materials  and 
testing  methods  2)  Benchmark  testing  results  3)  Benchmark  materials  and 
testing  methodology  validation  4)  Recommendations  for  benchmark  testing  of 
non-DMA  cartographic  data  capture  systems  5)  Recommendations  for  future 
research  and  development  and  6)  Observations  about  in-house  system 

characteristics  and  operating  procedures. 

1)  Benchmark  testing  materials  and  testing  methods  were  developed  during 
a  previous  DMA  research  effort  entitled  "A  Defense  Mapping  Agency  (DMA) 
Raster-to-Vector  Analysis".  The  benchmark  materials  consist  of  two 
basic  types: 

o  Three  sets  of  unique  cartographic  geometries  (i.e.,  simulated 
contours  -  concentric  circles,  drainage,  and  grids)  each  reproduced 
four  times  in  Increasing  levels  of  density 

o  Sample  DMA  products  Including  a  Digital  Terrain  Elevation  Data 

(DTED )  contour  and  drain/ridge  overlays,  a  DEAD  color  pencil 

compilation  and  a  Hydrographic  chart  compilation 

The  benchmark  testing  procedures  included: 

o  Scanning,  thinning,  vectorizing,  and  plotting  of  the  sixteen  input 
manuscripts.  Times  were  kept  for  the  first  three  procedures. 

o  Evaluation  criteria,  were  based  on  timings,  CRT  image  quality 

assessment,  digital  plot/analog  Input  "overlay"  analysis,  system 
integration/user  friendliness  evaluation,  and  numerical  analysis. 
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2)  Benchmark  testing  revealed  the  following  results: 

o  The  Broomall  AGOS  scanner  was  affected  by  the  amount  of  data  being 
processed  while  the  Scitex  was  not  affected. 

o  Data  density  adversely  affected  vectorization  times  for  the  AGDS 
particularly  for  the  synthetic  contour  and  grid  data.  Increasing 
data  density  does  not  appear  to  affect  the  Scitex  raster- to- vector 
conversion  rates  to  an  equal  degree.  In  fact,  conversion  rates  for 
the  synthetic  drainage  data  actually  improve  with  increasing  data 
density. 

o  The  AGDS  appeared  to  process  the  synthetic  contour  data  faster  than 
either  the  synthetic  drainage  or  grid  data,  although  not 
significantly  better.  The  Scitex  processed  the  synthetic  contour 
data  at  a  significantly  faster  rate  than  the  other  two  data  types, 
however. 

o  Overall  the  Scitex  performed  more  effectively  than  the  Broomall 
AGDS,  performing  three  to  four  times  faster. 

o  A  number  of  error  types  were  identified  in  the  visual  inspection  of 
digital  plots.  These  included:  gaps,  spikes,  slivers,  offsets,  and 
wandering  centerlines. 

3)  Generally,  the  benchmark  testing  materials  and  methodologies  were 

effective  in  evaluating  A/V  system  performance.  A  few  recommendations 

for  improvements  were  made: 

o  Maintain  separate  statistics  for  all  manual  or  interactive  editing 
required  during  the  benchmark  testing. 

o  Replace  the  hydrographic  sheet  in  the  benchmark  package  because  it 
does  not  represent  a  typical  hydrographic  compilation.  For 
example,  the  bathymetric  soundings  are  the  same  color  as  other 
pertinent  information  on  the  sheet. 

ill 


o  Develop  an  objectives  oriented  test  for  at  least  one  of  the  DMA 
sample  Inputs.  Require  data  capture  systems  to  produce  a  specified 
DMA  product  In  the  most  efficient  manner  possible.  For  example, 

this  might  entail  the  production  of  press-reac(y  color  separations 
for  a  hydrographic  chart  derived  from  a  color  compilation 
manuscript.  Timings  for  individual  procedures,  descriptions  of  the 
types  of  work  required  and  an  assessment  of  the  quality  of  the 
output  would  be  required. 

4)  Battel le  recommends  that  DMA  run  benchmark  tests  on  the  following 

commercially  available  cartographic  data  capture  systems: 

o  Scitex  Response- 280  -  Given  DMA's  current  utilization  of  Scitex 
technology  it  is  critical  that  ongoing  assessments  of  product 
improvements  be  performed. 

o  Intergraph  Scan  Data  Capture  System  -  Reports  of  new 

raster- to- vector  conversion  algorithms  and  hardware  processors 
make  this  newly  Introduced  system  an  ideal  candidate  for  benchmark 
testing. 

o  SYSSCAN  Kartoscan  -  In  depth  discussions  with  scientists  and  users 
of  this  system  have  revealed  a  state-of-the-art  data  capture 
system.  Continual  system  development  and  improvement  in  European 
and  U.S.  based  laboratories  make  this  an  attractive  system  for 
benchmark  testing.  Their  applications  development  with  DTED/DFAD 
type  data  in  Europe  is  another  good  reason  for  benchmark  testing. 

o  Laserscan  Lasertrak  -  This  system  is  recommended  because  it 

represents  a  unique  approach  to  cartographic  data  capture  via  laser 
line- foil  owing  technology.  It  may  be  particularly  effective  in  the 
capture  of  DTED  and  DFAD  data  at  DMA.  Its  recent  acquisition  at 
USGS  Is  another  incentive  for  benchmark  testing. 


5)  Battel  1 e  recommends  three  areas  for  future  research  and  development  at 
DMA: 

o  DMA  should  support  the  completion  of  the  qualitative/editing 

assessment  component  of  the  basic  benchmark  testing  package.  This 
component  consists  of  a  single  sheet  of  cartographic  geometries  in 
varying  degrees  of  geometric  degradation.  The  tests  for  this  input 
focus  on  system  generated  errors  and  the  automatic  functions  for 
correction  of  geometric  errors  on  input  data.  Battel le  recommends 
the  completion  of  this  test  material  and  the  validation  of  it  on 

the  DMAHTC  Scitex  and  AGDS  data  capture  systems 

o  Evaluation  of  state-of-the-art  cartographic  data  capture  systems 
indicates  that  important  progress  is  being  made  (by  commercial 
vendors)  in  the  areas  of  pattern  recognition,  feature  extraction, 
automatic  feature  tagging  and  spatial/topological  encoding. 

DMA  should  support  the  creation  of  an  enhanced  benchmark  testing 
capability,  based  upon  the  recently  developed  DMA  benchmark 

package,  which  addresses  these  forthcoming  advances. 

/ 

o  DMA  should  initiate  an  ongoing  program  of  in-house  commercial 

cartographic  data  capture  system  evaluation  and  upgrade. 

Specifically,  all  existing  software  routines  on  the  Scitex  Response 
-  250  should  be  catalogued  and  defined  in  terms  of  cartographic 
applications.  Additionally,  batch  processing  and  programming 

functions  on  the  system  should  be  seriously  investigated.  An 

assessment  should  also  be  performed  of  the  most  effective 

utilization  of  "manual"  interactive,  computer-assisted  and 

automatic  functions  on  the  Scitex. 

6)  During  the  benchmark  testing  of  the  Scitex  cartographic  data  capture 
system  a  number  of  observations  were  made  of  the  basic  system 
characteristics  and  current  operating  procedures.  Discussions  of 
these  observations  are  integrated  throughout  the  report. 
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RASTER-TO-VECTOR  BENCHMARK  TESTING 
1.0  INTRODUCTION 

Battelle's  Columbus  Laboratories  (BCL)  has  recently  completed 
the  benchmark  testing  of  two  automated  cartographic  data  capture  systems 
at  the  Defense  Mapping  Agency  Hydrographic/Topographic  Center  (DMAHTC) 
under  Contract  No.  DAAH01-83-D-A008,  Delivery  Order  No.  0030.  A  standard 
package  of  benchmark  testing  materials,  developed  during  a  previous  DMA 
sponsored  research  project*,  was  used  to  evaluate  analog-to- vector 
conversion  performance  of  the  Automated  Graphic  Digitizing  System 
(Broomall-AGDS)  and  Scitex  Response-250.  This  report  summarizes  the 
results  of  the  testing,  evaluates  the  utility  of  the  benchmark  materials 
and  testing  methodologies,  makes  observations  about  system  characteristics 
and  current  operating  procedures,  recommends  other  commercial  systems  for 
benchmark  testing,  and  points  out  areas  in  need  of  future  research  and 
development. 

A  number  of  observations  are  important  to  make  at  this  time. 
First,  neither  the  Scitex  Response  -250  nor  the  Broomall  AGDS  in  use  at 
DMAHTC  represents  the  most  current  state-of-the-art  cartographic  data 
capture  systems  in  today's  marketplace.  Scitex  no  longer  markets  or  sells 
the  Response  -  250.**  It  has  been  replaced  by  the  Response  -  280  (DMAAC 
has  recently  acquired  this  version  of  the  system).  The  Broomall  is  one  of 
the  original  systems  of  its  kind  and  many  technological  advances  have 
occurred  since  its  acquisition.  These  particular  systems  were  benchmark 
tested  due  to  their  accessibility  at  DMAHTC.  In  addition,  the  primary 
focus  of  the  testing  was  to  validate  the  benchmark  testing  materials  and 
testing  methodologies. 

A  second  comment  refers  to  the  utilization  of  benchmark 
testing  materials  developed  during  the  Defense  Mapping  Agency  (DMA) 
Raster-to-Vector  Analysis  project.  All  materials  were  used  in  the 
benchmark  testing  at  DMAHTC  except  the  synthetically  generated  quality 


Defense  Mapping  Agency  (DMA)  Raster-to-Vector  Analysis. 

The  Scitex  Response  -250  in  use  at  DMAHTC  was  running 
version  280  software  at  the  time  of  benchmark  tests. 
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assessment  sheet  (Synthetic  Test  Sheet  #1;  refer  to  the  DMA 

Raster- to-Vector  Analysis  project  final  technical  report*  for  a  full 

description  of  Its  contents).  Tine  restraints  and  a  longer  than 

anticipated  development  (of  the  synthetic  test  material)  period  prevented 
the  utilization  of  this  Input  to  the  benchmark  test. 

Thirdly,  the  benchmark  test  as  applied  to  the  cartographic 

data  capture  systems  at  DMAHTC,  addresses  only  Digitization  and 
Raster-to-Vector  Conversion  as  defined  In  the  DMA  Analog- to-Vector 
Conversion  Model  (see  DMA  Raster-to-Vector  Analysis  Final  Technical  Report 
for  definitions).  This  resulted  from  time  limitations  on  access  to 
production  equipment  at  DMAHTC.  Additionally,  the  benchmark  testing  of 
more  subjective  A/V  processing  functions  (e.g.,  manuscript  preparation. 
Interactive  data  editing  and  feature  tagging)  will  require  the  development 
of  more  sophisticated  human  factors  testing  methodologies. 

Sections  two  (2.0)  and  three  (3.0)  review  the  general 
characteristics  of  the  benchmark  testing  materials  and  the  testing 
methodologies.  (More  detailed  descriptions  are  presented  In  the  final 
technical  report  of  a  Defense  Mapping  Agency  (DMA)  ,Raster-To-Vector 
Analysis1  project.)  Sections  four  (4.0)  and  five  (5.0)  detail  the 
benchmark  test  results  for  the  AGDS  and  Sc Itex  systems,  respectively. 
Section  six  (6.0)  briefly  summarizes  a  comparison  of  benchmark  test 
results  between  the  AGDS  and  Scltex  systems.  Section  seven  (7.0) 
discusses  the  validity  of  the  benchmark  testing  materials  and  testing 
methodologies  and  makes  recommendations  for  Improvements.  Section  eight 
(8.0)  makes  specific  recommendations  for  non-DMA  cartographic  data  capture 
system  benchmark  testing.  Finally,  section  nine  (9.0)  outlines 
recommendations  for  future  research  and  development. 

2.0  CONCISE  REVIEW  OF  BENCHMARK  TESTING  MATERIALS 

There  are  two  basic  types  of  benchmark  testing  materials: 
Sample  DMA  Products  and  Synthetic  Test  Sheets. 

Sample  DMA  Products  consist  of  I)  a  drain/ridge  overlay  and  2) 

contour  sheet  (film  positives),  3)  a  color  pencil  compilation  of  a 

♦Selden,  David  D.,  Went,  Burton  H.,  Jr.,  and  Kleszczelskl ,  Stan  E., 

"Defense  Mapping  Agency  (DMA)  Raster-to-Vector  Analysis",  Report  No.  ETL  4, 
prepared  by  Battelle's  Columbus  Laboratories,  Tactical  Technology  Center, 
Columbus,  Ohio,  for  U.  S.  Army  Engineer  Topographic  Laboratories,  Mapping 
Developments  Division,  Contract  No.  DAAH01-83-D-A008,  MIPR  No.  3.13179 
(November  30,  1984). 
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hydrographic  chart,  and  4)  a  color  pencil  compilation  of  a  Digital  Feature 
Analysis  Data  (DFAD)  sheet.  The  tatter  two  samples  are  drafted  on  plastic 
Mylar.  All  products  were  chosen  to  represent  the  range  of  typical  DMA 
data  types,  data  densities  and  maMrlals. 

The  twelve  Synthetic  Test  Sheets  consist  of  three  types  of 
cartographic  geometries  {concentric*  -  highly  abstract  contours, 
simulated  drainage,**  and  orthogonal  grids)***  each  reproduced  four 
times  with  increasing  numbers  of  linear  inches.  This  second  set  of 
testing  materials,  generated  at  the  U.S.  Army  Engineer  Topographic 
Laboratories,  was  created  to  focus  on  geometric  patterns  typically  found 
on  DMA  products  to  determine  the  impact  of  geometry  and  increasing  data 
density  on  raster- to- vector  conversion  times.  Table  1A  on  page  four  (4) 
indicates  the  predetermined  length  in  linear  inches  of  each  of  the  twelve 
synthetic  input  sheets.  Although  an  attempt  was  made  to  generate  equal 
numbers  of  linear  inches  per  density  level  of  input,  algorithm  complexity 
prevented  complete  attainment  of  this  goal.  Rough  comparability  was 
achieved  between  all  four  density  levels  for  synthetic  contours  and 
synthetic  grids.  Levels  three  and  four  of  synthetic  drainage  are  roughly 
equivalent  to  density  levels  one  and  two  of  the  other  input  types. 

3.0  CONCISE  REVIEW  OF  BENCHMARK  TESTING  METHODOLOGIES 

Testing  methods  were  applied  equally  to  both  the  AGDS  and 
Scitex  systems  with  deviations  resulting  from  unique  system 
characteristics  or  limitations.  The  major  procedures  included:  raster 
scanning,  raster- to- vector  conversion,  and  film  plotting  of  the  vector 
data.  Automatic  and  "manual"  interactive  raster  editing  was  performed 
where  facilities  were  available  and  a  specific  need  identified.  The 
intent  of  the  processing  was  to  time  the  individual  steps  without  editing, 
however. 

*  Referred  to  as  SYNC0N1  thru  SYNC0N4  for  the  remainder 
of  the  report. 

**  Referred  to  as  SYNDRN1  thru  SYNDRN4  for  the  remainder 
of  the  report. 

***  Referred  to  as  SYNGRID1  thru  SYNGRID4  for  the 
remainder  of  the  report. 


Table  1A 


TWELVE  SYNTHETIC  TEST  MANUSCRIPTS 
LENGTH  IN  LINEAR  INCHES 


Oenslty 
Level  1 

Density 

Level  2 

Density 

Level  3 

Density 

Level  4 

SYNTHETIC  DRAINAGE 

540 

785 

1167 

1760 

(SYNDRN) 

SYNTHETIC  CONTOURS 

1267 

1647 

3729 

7717 

(SYNCON) 

SYNTHETIC  GRIDS 

1280 

1632 

3920 

7704 

(SYNGRID) 
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Evaluation  criteria  are  based  on  process  timing  (individual 
steps  and  combined),  virtual  image  quality  assessment,  digital  plot/analog 
input  "overlay"  analysis,  system  integration/user  friendliness  evaluation, 
and  numerical  analysis  of  timing  results.  A  complete  definition  and 
rationalization  of  this  set  of  criteria  is  presented  in  the  final  report 
of  the  Defense  Mapping  Agency  (DMA)  'Raster-to-Vector  Analysis'  project. 

'  4.0  A6DS  BENCHMARK  TESTING 

The  Broomall  AGDS  cartographic  data  capture  system  basically 
consists  of  a  large  format  flatbed  raster  scanner  ("black"  &  white),  a 
vectori2ation  subsystem  and  edit/tag  subsystem.  The  benchmark  test 
evaluated  those  functions  running  on  the  first  two  components  while  not 
addressing  the  interactive  edit/tag  subsystem  routines.  This  limitation 
was  imposed  due  to  time  restraints  in  addition  to  removing  from  the  test 
the  subjective  nature  of  Interactive  edit  functions  performed  by  human 
operators. 

Raw  data  for  raster  scanning  and  vectorization  times  is 
presented  in  Table  2A  (page  7).  Of  particular  note  is  the  apparent  impact 
of  data  density  on  scanning  time.  For  example,  SYNC0N1  contains  1267 
lineal  inches  and  required  one  hour  and  fifty-one  minutes  to  scan.  In 
comparison,  SYNC0N4  (of  equal  format  dimensions)  contains  7717  lineal 
inches  and  required  three  hours  and  forty-one  minutes  to  scan.  This  one 
hour  and  fifty  minute  discrepancy  may  be  attributable  in  part  to  the  way 
in  which  data  is  stored  in  a  scanning  buffer  and  written  to  disk  storage 
when  the  buffer  is  filled.  The  process  of  writing  (and  reportedly 
reformatting,  as  well)  slows.  If  not  stops,  the  forward  scanning  motion  of 
the  scanning  head.  The  more  often  this  filling  of  the  scanning  buffer  and 
writing  to  disk  storage  occurs  (obviously  increasing  with  greater  data 
density)  the  slower  the  scanning  appears  to  be.  Although  this  appears  to 
be  the  case  with  the  synthetic  contour  and  grid  sheets,  the  data  for 
synthetic  drainage  does  not  support  this  observation.  There  are  two 
possible  explanations.  The  specific  geometric  pattern  may  somewhat 
influence  this  process,  where  some  tend  to  slow  the  scanning  progress  more 
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than  others.  The  other  reason  may  reflect  the  setting  of  the  scanning 
limits.  Even  though  an  effort  was  made  to  set  the  scanning  size  limits 
and  scanning  parameters  equally  for  all  twelve  synthetic  test  sheets,  some 
unintended  variability  may  have  diminished  the  effect  for  the  synthetic 
drainage  sheets. 

4. 1  ASPS  Benchmark  Testing  Results 


Another  Important  trend  worthy  of  note  Is  the  steady  Increase 
In  times  for  vectorlzatlon  as  the  number  of  linear  Inches  Increases.  This 
was  fully  anticipated,  logic  dictating  an  Increased  processing  time  for 
greater  amounts  of  data.  Of  greater  Interest  Is  the  raster- to- vector 
linear  Inches  per  minute  conversion  rates.  Table  3A  (page  8)  presents  the 
linear  Inch  per  minute  conversion  rates  for  all  twelve  synthetic  test 
sheets.  Please  note  the  precipitous  drop  In  rates  for  synthetic  contour 
and  grid  data  conversion  as  data  densities  Increase.  Conversely,  there  Is 
a  slight  overall  improvement  in  the  conversion  rate  for  the  four  synthetic 
drain  inputs.  This  points  to  the  increased  Inefficiencies  of  the  current 
vectorization  algorithms  with  Increasing  amounts  of  data,  particularly  for 
simulated  contour  and  grid  geometries.  The  apparent  steady  rate  for 
synthetic  drainage  conversion  m*y  be  somewhat  misleading.  The  range  in 
linear  Inches  for  this  data  type  Is  only  1220*  Inches.  The  range  for 
the  other  two  data  types  Is  6450**  inches.  Table  4A  (page  8.1)  gives  a 
better  view  of  how  the  conversion  rates  compare  for  all  three  data  types 
for  comparable  data  densities.  It  appears  that  the  AGDS  converts  the 
simulated  contour  data  slightly  more  efficiently  than  the  other  data 
types.  This  may  reflect  the  system's  original  design  for  processing 
contour  data.  On  the  other  hand,  the  heaviest  data  level  Indicates  little 
difference  between  the  grldded  and  non- Intersecting  concentric  data  Input 
conversion  rates. 

*  This  number  derives  from  subtracting  540  (number 
of  linear  Inches  for  SYNDRN1)  from  1760  (number 
of  linear  Inches  for  SYNDRN4). 

**  This  number  derives  from  subtracting  either  1267 
or  1280  (number  of  linear  Inches  for  SYNC0N1  and 
SNYGRID1)  from  7717  or  7704  (number  of  linear 
Inches  for  SYNC0N4  and  SYNGR1D4),  respectively. 
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Table  2A 

AGDS  BENCHMARK  TESTING 
RAW  DATA  RESULTS 


INPUT  I 

t  LINEAL  INCHES 

SCAN  TIME 

VECTORIZATION  TIME 

KR/MIN 

HR/MI N 

SYNDRN1 

540 

01:54 

00:28 

SYNDRN2 

785 

01:41 

00:42 

SYNDRN3 

1167 

01:50 

00:52 

SYNDRN4 

1760 

01:41 

01:25 

SYNC0N1 

1267 

01:51 

00:59 

SYNC0N2 

1647 

02:00 

01:11 

SYNCON3 

3929 

02:09 

03:48 

SYNCON4 

7717 

03:41 

12:54 

SYNGRID1 

1280 

- 

01:04 

SYNGRID2 

1632 

02:00 

01:04 

SYNGRID3 

3920 

02:05 

05:29 

SYNGRID4 

7704 

03:01 

13:05 

CONTOUR  (DMA) 

01:55 

06:01 

DRN/RDG  (DMA) 

01:35 

01:12 

HYDRO  (DMA)* 

01:52 

03:49 

DFAD  (DMA) 

01:16 

02:52 

*  1/2  Compilation  sheet  was  processed, 


Tib]«  3A 

AGDS  -  VECTOR I ZAT ION  RATES 


INPUT 

«  LINEAL  INCHES 

INCHES/MINUTE 

SVNORN1 

540 

19.29 

SYN0RN2 

785 

18.69 

SYICRN3 

1157 

22.44 

SVNDRN4 

1760 

20.71 

SYNC ON I 

1267 

21.47 

SYNC0N2 

1647 

23.20 

SYNC0N3 

3929 

17.23 

SYNC0N4 

7717 

9.97 

SYNGR101 

1280 

20.00 

SYNGRID2 

1632 

17.36 

SYNGRI03 

3920 

11.91 

SYNGRI04 

7704 

9.81 

Tibi*  4A 

AGOS  -  COMPARISON  OF  BATA  CONVERSION 
RATES  FOR  SYNTHETIC  GEOMETRIES  OF  COMPARABLE  DENSITIES 


SVNDRN3 

SVNC0N1 

SYNGR1D1 

f  INCHES 

1167 

1267 

1280 

INCHES/MIN. 

22.44 

21.47 

20.00 

SYNDRN4 

SYNC0N2 

SYNGRID2 

1  INCHES 

1760 

m 

1632 

INCHES/MIN. 

20.71 

23.20 

17.36 

SYNC0N3 

SYNGR1D3 

«  INCHES 

3929 

3920 

INCHES/MIN. 

17.23 

11.91 

SYNC0N4 

SYNGR104 

f  INCHES 

2211 

7704 

INCHES/MIN. 

hSL 

9.81 

malitative  Assessment  of  AGDS  Benchmark  Test  Resul ts 


This  qualitative  assessment  is  based  primarily  on  observations 
made  of  digital  plots  of  vector  data  overlaid  on  the  original  input 
manuscript.  Although  each  input  was  individually  processed  and  plotted, 
for  the  sake  of  clarity  and  conciseness  this  discussion  will  summarize  the 
results  of  each  group  of  synthetic  input.  Examples  of  errors  and 
anomalies  are  presented  in  Figures  1A  and  IB  (pages  11-12).  (Please  note 
that  all  AGDS  data  was  automatically  passed  through  a  point  filter  and 
spike  removal  routine  prior  to  plotting.) 

Although  there  are  few  options  or  parameters  to  set  on  the 
Broomall  raster  scanner  or  vectorization  subsystems,  one  observation  can 
be  made  about  their  current  utilization.  It  appears  that  a  fixed  group  of 
parameter  settings  has  been  developed  and  they  a  re  generally  applied  without 
concern  for  the  unique  characteristics  of  input  data.  In  fact,  little  is 
known  about  the  impact  of  changing  these  settings.  This  rigid  application 
of  parameters  may  result  in  greater  numbers  of  errors  (e.g.,  scanner 
induced)  and  conversely,  in  the  diminished  success  of  the  [THN,  INODES, 
THN]  routines  on  the  vectorizer.  (THN  removes  data  stubs  and  fills  in 
holes  in  data.  INODES  reduces  the  number  of  points  defining  lines.) 

4.2.1  Synthetic  Drainage  Evaluation  (AGDS) 

All  four  synthetic  drainage  plots  were  accurately  scaled 
although  limited  linear  misalignments  were  noted  on  SYNDRN3  and  SYNDRN4. 
The  only  obvious  errors  were  found  in  the  form  of  minor  line  breaks  on 
SYNDRN2.  It  is  possible  these  were  caused  by  plotter  skipping  and  not 
missing  data.  (This  points  to  the  advantages  of  film  plotting  for  quality 
assurance  tests).  Generally,  the  synthetic  drainage  plots  were  of  very 
high  quality. 

4.2.2  Synthetic  Contour  Evaluation  (AGDS) 

All  four  synthetic  contour  plots  were  accurately  scaled 
although  some  linear  misalignments  were  found  on  SYNC0N2  and  SYNC0N4. 


These  consist  of  one  to  two  llnewelght  offsets  from  the  center  of  the 
analog  Input  or  a  general  wavy  appearance.  Only  a  minimum  number  of  line 
gaps  were  In  evidence  and  a  single  errant  line  was  noted  on  SYNC0N4. 
SYNC0N3  and  SYNC0N4  also  exhibited  straight  edge  perimeters  for  their 
center  "circles."  On  the  whole,  all  four  synthetic  contour  plots 
exhibited  a  good  quality. 

4.2.3  Synthetic  Grid  Evaluation  (AGOS) 

All  four  synthetic  grid  plots  were  accurately  scaled. 
However,  linear  misalignments  and  Intersection  offsets  were  visible  on 
every  plot.  Some  breaks  In  lines  were  noticed  on  SYNGRID4.  This  sheet 
also  exhibited  the  most  significant  Intersection  offsets.  Again,  this  may 
reflect  a  plotter  difficulty,  particularly  for  short  line  segments  on  the 
densest  Input  sheet.  Overall  plotted  data  quality  was  good. 

4.2.4  Digital  Feature  Analysis  Data  (DFAD)  Evaluation  -  (AGDS) 

The  digital  plot  was  accurately  scaled  to  the  analog  Input. 
Lines  appeared  to  he  smooth  although  some  "unnecessary"  squlggles,  offsets 
and  misalignments  were  noted.  Occasional  line  breaks  were  also  found. 
These  appeared  to  be  more  than  plotter  generated  and  may  reflect  scanner 
Incapability  to  capture  "weak”  Input  lines.  General  quality  was 
considered  good. 

4.2.5  Contour  and  Drain/Ridge  Overlay  Evaluation  -  (AGDS) 

Both  the  contour  and  drain/ridge  digital  plots  exhibited  a 
slight  scaling  problem.  Despite  this,  good  linear  alignment,  even  In 
dense  areas  was  noted.  Few  or  no  errors  or  other  anomalies  were 
noticed.  Some  coalescence  of  contours  In  dense  areas  was  found.  Overall 
quality  was  considered  good. 


Figure  1A 


Digital 

vector 

average 

larger. 


ERRORS  AND  ANOMALIES  EXHIBITED  ON 
DIGITAL  PLOTS  OF  BENCHMARK  TESTING 
MATERIALS  -  (AGDS)* 

LINEAR  MISALIGNMENT  (SIMULATED  BLOWUP) 


Digital  Vector  Plot 

n^i 


Analog  Input 


WANDERING  CENTERLINE  (SIMULATED  BLOWUP) 


Digital  Vector  Plot 

id. 


Analog  Input 


OFFSET  LINES 


plots  were  plotted  on  a  XYNETICS  ballpoint 
plotter.  Misalignments  measured  on  the 
.003"-. 004",  Occasional  gaps  were  somewhat 
in  the  range  of  .01". 


** . 
hi 


Figure  IB 


ERRORS  AND  ANOMALIES  EXHIBITED  ON 
DIGITAL  PLOTS  OF  BENCHMARK  TESTING 
MATERIALS  -  (AGDS) 


MISALIGNED  INTERSECTIONS 


•DIGITAL  GAPS" 


UNNECESSARY  "SQUIGGLES" 
ON  GRIDLINES 


(SIMULATED  BLOWUP) 
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4.2.6  Hydrographic  Chart  Overlay  Evaluation  -  (AGDS) 

A  definite  scaling  problem  was  evidenced.  All  red  lines  on 
the  input  sheet  were  also  dropped  (due  to  a  red  light  scanning  laser). 
This  obviously  represents  a  problem  with  scanning  color  coded  manuscripts 
on  the  AGDS.  All  bathymetric  soundings  were  also  vectorized.  The  AGDS 
does  not  provide  any  raster  editing  tools  which  could  be  applied  for 
selected  data  removal  (including  the  numbers).  All  colors  but  red  were 
captured,  vectorized,  and  plotted.  Thus,  it  is  not  quite  correct  to  refer 
to  the  scanner  on  the  AGDS  as  a  "black  and  white"  scanner,  at  least  not 
strictly  speaking.  On  the  other  hand  it  does  not  provide  color  separation 
scanning  capabilities  either. 

5.0  SCITEX  BENCHMARK  TESTING 

The  Scitex  Response-250  cartographic  data  capture  system 
consists  of  a  large  format  drum  color  raster  scanner,  a  raster 
colorediting  station  and  a  large  format  laser  film  plotter.  The  benchmark 
tests  utilized  all  three  components  of  the  system,  concentrating  on 
scanning,  thinning,  vectorization  times,  data  anomalies  and  overall  system 
performance.  The  benchmarks  were  run  over  a  period  of  several  months  and 
utilized  all  available  system  components  when  available. 

5. 1  Scitex  Benchmark  Testing  Results 

Raw  data  for  raster  scanning  and  vectorization  times  is 
presented  in  Table  5A  (page  16).  The  raw  data  sheet  for  the  Scitex 
provides  an  extra  column  of  information,  thin  time  (as  compared  to  the 
AGDS)  which  is  indicative  of  the  two  step  raster-to-vector  conversion 
procedure  implemented  on  this  system.  Please  note  the  fairly  consistent 
scanning  times  irrespective  of  data  density.  Overall,  thin  times  appear 
to  be  unaffected  by  increasing  data  density  for  both  the  synthetic 
drainage  (SYNDRN)  and  synthetic  grid  (SYNGRID)  inputs.  This  is  not  the 
case  for  the  synthetic  contour  data  (SYNCON),  where  increasing  data 
density  results  in  longer  thin  times.  In  comparison,  we  see  a  marked 
increase  in  vectorization  times  paralleling  increasing  data  densities. 
This  applies  equally  for  all  three  synthetic  inputs. 
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Table  6A  (page  17)  presents  thinning,  vectorizing,  and 
combined  thin/vectorization  linear  inches  conversion  rates  for  the 
Scitex.  Thin  conversion  rates  improve  with  increasing  data  density  for 
synthetic  drainage  and  synthetic  grid  input  data.  The  synthetic  contour 
data  thin-conversion  rates  do  not  perform  in  exactly  the  same  manner. 
Although  SYNC0N3  and  SYNC0N4  demonstrate  improving  rates,  they  are  not 
attaining  efficiencies  higher  than  SYNC0N1  which  is  the  case  with  the 
synthetic  drainage  and  grid  data  thin-conversion  rates.  It  should  be 
noted  that  the  thin  conversion  rates  for  the  synthetic  contours  and  grids 
are  significantly  better  than  those  attained  for  drainage  data. 

Combined  thin  and  vectorization  times  provide  some  interesting 
contrasts.  The  synthetic  drainage  data  conversion  rates  steadily  improve 
with  increasing  data  density.  Synthetic  contour  data  demonstrates  a  more 
constant  conversion  rate  performance  although  large  data  input  does  slow 
the  processs  somewhat.  The  synthetic  grid  data  conversion  rates  overall 
are  better  than  the  synthetic  drainage  rates  and  curiously  the  fourth 
level  of  data  shows  the  greatest  success.  The  synthetic  contour  combined 
conversion  rates  are  significantly  better  than  either  of  the  other  two 
data  types. 

Table  7A  (page  18)  presents  a  more  realistic  view  of  how  the 
combined  conversion  rates  compare  for  all  three  data  types  for  comparable 
data  densities.  It  appears  that  for  all  four  levels  of  data  density,  the 
Scitex  converts  the  synthetic  contour  data  most  efficiently.  This  appears 
to  reflect  on  the  relative  difficulties  of  converting  intersecting  and 
merging  (or  intersecting  and  crossing)  data  as  compared  to 
non- intersecting  data. 

5 . 2  Qualitative  Assessment  of  Scitex  Benchmark  Test  Results 

This  qualitative  assessment  is  based  primarily  on  observations 
made  of  digital  raster/vector  data  and  digital  plots  of  raster*  data 
overlaid  on  the  original  input  manuscript.  Even  though  each  input  was 
individually  processed,  only  select  samples  of  each  of  the  synthetic  input 
data  types  were  actually  plotted.  The  observations  in  the  following 
sections  will  address  each  group  of  synthetic  input  data  types  in  summary 


fashion.  Examples  of  errors  and  anomalies  observed  on  CRT  screens  or 
digital  plots  are  presented  in  Figures  2A  and  2B  (pages  21-22).  Please 
note  that  all  Scitex  data  were  processed  with  minimal  editing  performed. 
Such  editing  was  performed  only  where  continued  processing  required 
limited  data  modification  (e.g.,  removal  of  tape  marks  from  raster  data 
prior  to  thinning).  In  addition,  all  digital  raster  plots  evidenced  a 
scaling  offset.  This  was  traced  to  a  lack  of  precise  calibration  on  the 
Scitex  raster  scanner.  Another  issue  worthy  of  note  is  the  apparent 
sensitivity  of  the  Scitex  conversion  (R/V )  algorithms.  For  example,  on 
numerous  occasions,  a  vectorization  failed  apparently  due  to  the  existence 
of  a  limited  number  of  unthinned  vectors  in  the  file.  The  existence  of 
fat  lines  or  tape  marks  in  a  raster  data  file  often  greatly  extended 
thinning  times  with  often  less  than  satisfactory  results  (i .e.,  unthinned 
lines  remained). 


*  The  plotting  of  digital  raster  data  was  not  originally 
anticipated  for  proof  plotting  during  the  benchmark 
testing.  Attempts  were  made  to  take  into  account  the 
unique  characteristics  of  raster  data  in  making  quality 
assessments. 
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Table  6A 


SCITEX  RASTER- TO- VECTOR  CONVERSION  RATES 
THINNING,  VECTORIZING,  COMBINED  TIMINGS 


INPUT 

#  LINEAL  INCHES 

INCHES  THINNED/MIN. 

INCHES  VEC./MIN. 

THIN-VECT. 

INCHES/MIN 

SYNDRN1 

540 

24.55 

45.00 

15.88 

SYNDRN2 

785 

31.40 

56.07 

20.13 

SYNDRN3 

1167 

53.05 

58.35 

27.79 

SYNDRN4 

1760 

70.40 

48.89 

28.85 

SYNCON1 

1267 

211.17 

79.19 

57.59 

SYNCON2 

1647 

126.69 

82.35 

49.91 

SYNCON3 

3929 

135.48 

80.18 

50.37 

SYNC0N4 

7717 

208.57 

69.52 

52.50 

SYNGRID1 

1280 

91.43 

53.33 

33.68 

SYNGRID2 

1632 

77.71 

81.60 

39.80 

SYNGRID3 

3920 

196.00 

56.00 

43.56 

SYNGRID4 

7704 

385.20 

38.52 

35.02 

Table  7A 


SCITEX  -  COMPARISON  OF  COMBINED  DATA 
CONVERSION  RATES  FOR  SYNTHETIC  GEOMETRIES 
OF  COMPARABLE  DENSITIES 


SYNDRN3 

SYNC0N1 

S YNGRID1 

IINCHES 

1167 

1267 

1280 

INCHES/MIN. 

27.79 

57.59 

33.68 

SYNDRN4 

SYNC0N2 

SYNGRID2 

#  INCHES 

1760 

1647 

1632 

INCHES/MIN. 

28.85 

49.91 

39.80 

SYNC0N3 

SYNGRID3 

#  INCHES 

3929 

3920 

INCHES/MIN 

50.37 

43.56 

SYNC0N4 

SYNGRID4 

#  INCHES 

7717 

7704 

INCHES/MIN. 

52.50 

35.02 
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5.2.1  Synthetic  Drainage  Evaluation  (Scltex) 

The  densest  level  synthetic  drainage  data  (SYNDRN4)  was 
plotted  on  the  Scltex  raster  plotter.  Other  than  the  overall  scaling 
problem,  an  anomaly  was  Identified  at  all  the  Intersections  of  the 
drainage  segments.  A  rounding  or  squaring  of  the  Intersection  of  three 
line  segments  was  prevalent  throughout  the  plotted  manuscript.  Line 
alignment  was  generally  good  In  all  other  Instances. 

5.2.2  Synthetic  Contour  Evaluation  (Scltex) 

Line  quality  appeared  to  be  smooth  and  representative  of  the 
analog  Input.  No  errors  or  anomalies  were  observed.  The  center-most 
circle  was  smooth  and  exhibited  no  squaring  of  the  circumference  similar 
to  the  AGDS  synthetic  contour  plots.  This  was  considered  a  high  quality 
ouput. 


5.2.3  Synthetic  Grid  Evaluation  (Scltex) 

Synthetic  grid  level's  two  (SYNGRID2)  and  three  (SYNGRID3) 
were  plotted  on  the  Scltex  raster  plotter.  In  addition  to  the  overall 
sheet  scaling  (approximately  .06"  in  one  axis),  offsetting  “linear"  lines 
were  observed  throughout  the  plotted  sheets.  More  than  likely  this 
reflects  the  vagaries  of  the  thinning  process  where  the  algorithm  appears 
to  be  only  accurate  to  plus  or  minus  one  unit  of  resolution  on  either  side 
of  a  centerline.  The  Inconsistency  of  the  offset  results  In  unsmooth 
lines.  A  second  problem  encountered  on  the  digital  plot  was  unthinned 
lines.  Although  rare,  occasional  Instances  of  multi -pixel -wide  lines  were 
Identified.  This  may  have  resulted  from  variations  In  the  width  of  input 
analog  lines.  In  these  cases  thinning  failed  to  reduce  the  lines  to  one 
unit  of  resolution.  A  third  problem  was  Identified  on  SYNGRID3. 
Instances  of  slivers  were  noted:  double  lines  resulting  from  single  line 
Input.  In  this  case  the  slivers  may  be  a  result  of  scanner  or  thinning 
difficulties. 


5.2.4  Digital  Feature  Analysis  Data  (DFAD)  Evaluation  -  (Scltex) 

Visual  observation  of  the  digital  plot  revealed  a  series  of 
errors  and  anomalies.  These  Included  stubs,  donuts,  unthinned  lines,  and 
some  evidence  of  wandering  centerlines.  Much  of  this  Is  attributable  to 
the  color  pencil  Mylar  Input  and  the  thinning  process.  It  Is  conceivable 
that  further  adjustments  to  the  scanning  calibration  might  eliminate  some 
of  these  anomalies. 

5.2.5  Contour  and  Drain/Ridge  Overlay  Evaluation  -  (Scltex) 

Digital  plots  were  not  available  soon  enough  for  Battelle  to 
perform  an  overlay  analysis.  However,  according  to  DMAHTC  staff,  visual 
observation  of  these  plots  revealed  a  "good  quality"  output. 

5.2.6  Hydrographic  Chart  Overlay  Evaluation  -  (Scltex) 

Smooth,  high  quality  lines  were  observed  on  the  digital  plot. 
Occasional  stubs  were  Identified.  It  should  be  noted  that  the  particular 
plot  reviewed  had  been  Interactively  edited  frame  by  frame  on  the  Scltex 
raster  editing  station  prior  to  plotting. 


Figure  2A 


ERRORS  AND  ANOMALIES  EXHIBITED  ON 
OIGITAL  PLOTS  OR  CRT  IMAGES  OF 
BENCHMARK  TESTING  DATA  -  (SCITEX)* 

THREE  SEGMENT  INTERSECTION  ANOMALY 


Correct  Rounded 

Representation  Anomaly 


OFFSETTING  LINEAR  LINES  (.003  -  .0 


Digital  plots  were  produced  on  thescitex 
laser  plotter. 


Figure  2B 


ERRORS  AND  ANOMALIES  EXHIBITED  ON 
DIGITAL  PLOTS  OR  CRT  IMAGES  OF 
BENCHMARK  TESTING  DATA  -  (SC I TEX) 
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STUBS  (Post  Thinning  Results) 

(Average  stub  length  - 
3  to  6  pixels  at  20  points/mm) 


HOOK  STUB 


V  STUB 


STRAIGHT  STUB 


CIRCULAR  STUB 
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6.0  COMPARISON  OF  BENCHMARK  TESTING  RESULTS 
SCITEX  RESPONSE-250  AND  BROOMALL  AGDS 


Several  charts  on  the  following  pages  list  the  various  test 
results  for  the  two  systems  side  by  side.  Tables  8A  and  9A  (pages  24-25) 
compare  first,  the  raw  scanning  times,  and  second.  the  combined 
raster-to-vector  conversion  times  for  the  Scitex  and  AGDS.  Figures  3A  and 
4A  (pages  26-27)  present  graphs  of  these  data.  It  appears  that  the  Scitex 
scan  times  are  more  constant  and  less  subject  to  influence  by  increasing 
data  density.  The  Scitex  appears  to  be  somewhat  faster  overall.  For 
example,  the  average  scan  time  for  the  twelve  synthetic  test  sheets  was 
two  hours  and  ten  minutes  (2:10)  for  the  Scitex  as  compared  to  two  hours 
and  thirty-three  minutes  (2:33)  for  the  AGDS.  Comparison  of  total 
raster-to-vector  conversion  times  demonstrates  a  dramatic  advantage  by  the 
Scitex.  In  some  cases,  this  advantage  approaches  almost  four  to  one  (e.g. 
note  12:54  hour  AGDS  vectorization  time  for  SYNC0N4  as  compared  with  2:28 
hours  combined  thin  and  vectorization  time  on  the  Scitex).  Average 
vectorization  time  on  the  AGDS  for  the  twelve  synthetic  inputs  was  three 
hours  and  fifty-eight  minutes  (3:58)  as  compared  to  one  hour  and  eighteen 
minutes  (1:18)  on  the  Scitex.  This  advantage  clearly  extends  to  the  DMA 
sample  data  as  well.  The  Scitex  conversion  times  for  the  contour, 
drain/ridge  and  DFAD  data  are  on  the  order  of  three  and  four  to  one 
improved  over  the  AGDS. 

Table  10A  (page  28)  presents  a  comparison  of  raster-to-vector 
conversion  rates.  Figure  5A  (page  29)  presents  a  graph  of  the  compared 
vectorization  rates.  Both  serve  to  re-emphasize  the  relative 
effectiveness  of  the  Scitex  compared  to  the  AGDS.  Rates  are  significantly 
higher  for  the  Scitex,  particularly  for  the  synthetic  contour  and  grid 
data.  The  average  conversion  rate  for  the  twelve  synthetic  inputs  on  the 
Scitex  is  37.92  inches/minute  compared  to  17.67  inches/minute  for  the 
AGDS. 

In  general,  it  appears  that  the  Scitex  Response-250  at  DMAHTC 
performed  significantly  better  than  the  Broomall  AGDS  when  tested  using 
equal  data  input.  Overall  conversion  times  and  rates  re-emphasized  this 


Table  8A 

COMPARISON  OF  SCAN  TIME 
SCITEX  AND  AGDS 


INPUT  # 

LINEAL  INCHES 

AGDS  SCAN  TIME 

SCITEX  SCAI 

HR/MIN 

HR/MIN 

HR/MIN 

SYNDRNl 

540 

01:54 

01:42 

SYNDRN2 

785 

01:41 

01:41 

SYNDRN3 

1167 

01:50 

01:42 

SYNDRN4 

1760 

01:41 

01:43 

SYNCON1 

1267 

01:51 

01:43 

SYNC0N2 

1647 

02:00 

01:42 

SYNCON3 

3929 

02:09 

01:38 

SYNC0N4 

7717 

03:41 

01:44 

SYNGRID1 

1280 

- 

01:52 

SYNGRID2 

1632 

02:00 

02:00 

SYNGRID3 

3920 

02:05 

01:50 

SYNGRID4 

7704 

03:01 

01:58 

CONTOUR  (DMA) 

01:55 

01:05 

DRN/RDG  (DMA) 

01:35 

01:20 

HYDRO  (DMA) 

01:52 

03:30 

DFAD  (DMA) 

01:16 

01:40 
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Table  9A 

COMPARISON  OF  COMBINED  RASTER-TO-VECTOR 
CONVERSION  TIMES  -  SCITEX  AND  AGDS 


INPUT 

TIME 

#  LINEAL  INCHES 

SYNDRN1 

540 

SYNDRN2 

785 

SYNDRN3 

1167 

SYNDRN4 

1760 

SYNC0N1 

1267 

SYNC0N2 

1647 

SYNC0N3 

3929 

SYNC0N4 

7717 

SYNGRID1 

1280 

SYNGRID2 

1632 

SYNGRID3 

3920 

SYNGRID4 

7704 

CONTOUR  (DMA) 

DRN/RDG  (DMA 

HYDRO  (DMA) 

DFAD  (DMA) 

AGDS  VECTOR  TIME 
HR/MIN 

SCITEX  THIN/VECT 
HR/MIN 

00:28 

00:34 

00:42 

00:39 

00:52 

00:42 

01:25 

01:01 

01:59 

00:22 

01:11 

00.33 

03:48 

01:18 

12:54 

02:28 

01:04 

00:38 

01:34 

00:41 

05:29 

01:30 

13:05 

03:40 

06:01 

02:30 

01:12 

00:37 

03:49 

00:67 

02:53 


00:41 


Scan  Time 

Synthetic  Geometries 


Contours 


Contours 


Table  10A 


COMPARISON  OF  COMBINED  RASTER-TO- VECTOR 
CONVERSION  RATES  -  SCITEX  AND  AGDS 


INPUT 

#  LINEAL  INCHES 

AGDS 

SCITEX 

INCHES/MINUTE 

INCHES/MINUTE 

SYNDRN1 

540 

19.29 

15.88 

SYNDRN2 

785 

18.69 

20.13 

SYNDRN3 

1167 

22.44 

27.79 

SYNDRN4 

1760 

20.71 

28.85 

SYNCON1 

1267 

21.47 

57.59 

SYNC0N2 

1647 

23.20 

49.91 

SYNCON3 

3929 

17.23 

50.37 

SYNC0N4 

7717 

9.97 

52.50 

SYNGRID1 

1280 

20.00 

33.68 

SYNGRID2 

1632 

17.36 

39.80 

SYNGRID3 

3920 

11.91 

43.56 

SYNGRID4 

7704 

9.81 

35.02 

Raster-to-Vector  Conversion  Rates 
Synthetic  Geometries 


Contours 


consistently  for  both  the  synthetic  and  DMA  sample  benchmark  testing 
data.  It  should  be  noted  that  the  digital  plots  revealed  some  qualitative 
advantage  demonstrated  by  the  Broomall  AGDS.  However,  this  Is  mitigated 
by  the  fact  that  all  data  In  this  system  passed  through  digital  filters 
and  spike/stub  removal  routines  (INODES  and  THN)  prior  to  plotting  as  a 
standard  operating  procedure.  This  was  not  the  case  on  the  Scltex  system. 

7.0  ASSESSMENT  OF  BENCHMARK  TESTING 
MATERIALS  AND  TESTING  METHODS 

One  of  the  objectives  of  running  benchmark  tests  on  the  DMAHTC 
Scltex  and  AGDS  was  to  validate  the  benchmark  materials  and  testing 
methodologies.  Much  was  learned  about  the  utility  of  the  specific  tests 
and  the  materials  used  resulting  In  an  overall  assessment  of  the  benchmark 
testing  package  Including  a  few  recommendations  for  Improvements. 

7.1  Benchmark  Testing  Materials 


Generally,  the  current  set  of  benchmark  materials  Is 
satisfactory  for  evaluating  cartographic  data  capture  systems.  The  twelve 
synthetic  test  sheets  provide  Important  Information  about  a  system's 
raster- to-vector  conversion  strengths  and  weaknesses  vis-a-vis  different 
cartographic  geometries  and  Increasing  data  densities.  It  provides  a 
basic  gauge  for  assessing  performance  levels  and  assists  DMA  In  predicting 
productivity.  The  DMA  sample  materials  provide  an  opportunity  to  assess  a 
system's  capability  for  assimilating  typical  analog  Input.  The  DFAD  sheet 
appears  to  represent  a  typical  sheet  as  do  the  contour  and  drain/ridge 
sheets.  A  problem  Is  perceived  with  the  current  DMA  Hydrographic  Chart 
compilation  example  In  the  benchmark  package.  It  Is  Battelle's 
understanding  that  most  chart  compilations  maintain  bathymetric  soundings 
In  a  unique  color.  This  particular  sheet  has  the  soundings  and  other 
pertinent  Information  In  the  same  color  (black).  This  requires  either  the 
vectorlzatlon  of  the  soundings  (non-standard  procedure)  or  the  elimination 
of  all  "black  data"  prior  to  vectorlzatlon,  or  the  step  by  step  manual 


elimination  of  soundings  (very  time  consuming).  It  Is  Battel le's 
understanding  that  such  a  compilation  would  he  rejected  hy  the  Digital 
Hydrography  Section  under  normal  production  operations.  These  facts 
should  he  taken  Into  consideration  hy  DMA  prior  to  using  the  hydrographic 
sheet  In  further  benchmark  testing.  Replacement  of  this  sheet  with  a  more 
representative  example  Is  one  possible  option. 

7.2  Benchmark  Testing  Methods 

Battel le  feels  that  the  timing  statistics  (raw  numbers  and 
derived  rates)  are  useful  and  valid  Indicators  of  system  performance 
capabilities.  The  scanning,  thinning  and/or  vectoring,  and  plotting  of 
data  Is  a  basic  procedure  for  benchmark  testing.  It  should  continue  to  be 
viewed  as  such.  These  procedures  as  applied  to  the  twelve  synthetic 
testing  materials  during  the  benchmarking  at  DMA  are  sound.  However,  one 
recommendation  Is  that  separate  statistics  be  maintained  for  all  manual 
and  Interactive  editing  of  synthetic  Input  data.  (Please  remember  that 
editing  is  not  a  prescribed  procedure  for  the  synthetic  benchmark 
testing). 

Timing  statistics  for  processing  of  DMA  sample  products  are 
considered  useful  and  valid  as  Indicators  of  system  performance.  More 
thought  needs  to  be  given  to  the  purpose  of  using  standard  DMA  materials. 
One  recommendation  Is  that  an  objectives  oriented  test  be  developed  for  at 
least  one  of  the  sample  Inputs.  This  means  that  DMA  should  establish  a 
specified  output  requirement  for  a  particular  testing  material.  A 
cartographic  data  capture  system  should  be  applied  towards  achieving  the 
specified  standard  In  the  most  efficient  manner.  Individual  process  and 
combined  processing  timings  should  be  kept.  Each  process  should  be 
categorized  as  a)  human  manual  b)  computer  Interactive  c)  computer- 
assisted  and  d)  automatic.  This  will  provide  DMA  with  an  understanding  of 
the  personnel,  task  type  end  time  requirements  to  produce  a  typical  DMA 
output  which  meets  acceptable  quality  standards.  An  example  of  one 
objective  oriented  benchmark  test  would  be  to  require  a  vendor  to  produce 
press  ready  color  separations  of  a  Hydrographic  Chart  derived  from  a 


standard  compilation  manuscript.  Time,  quality  and  activity  types  Mould 
comprise  the  basic  evaluation  criteria.  Another  example  Mould  be  a 
requirement  for  a  vendor  to  produce  an  elevation  matrix  suitable  for  DTED 
cell  generation.  Again,  time,  quality  and  activity  types  Mould  be  used  to 
assess  system  performance  levels. 

Another  Issue  has  been  raised  concerning  the  number  of 
benchmark  testing  materials  (sixteen)  currently  In  the  DMA  testing 
package.  The  concern  has  been  expressed  that  perhaps  too  many  Inputs  Mill 
overburden  prospective  vendors  of  cartographic  data  capture  systems. 
Battel le  does  not  believe  this  to  he  the  case.  First,  any  neM 
state-of-the-art  system  to  be  benchmarked  should  match  and  probably 
surpass  the  performance  levels  of  the  Scltex  Response-250.  Thus  time 
required  on  these  systems  should  diminish.  It  Is  Interesting  to  note  that 
total  actual  processing  time  on  the  Broomall  AGDS  Mas  approximately 
eighty-eight  hours  and  only  forty-eight  hours  on  the  Scltex.  (This  does 
not  Include  plotting  and  represents  a  summation  of  final  successful  runs, 
and  thus  no  re-runs.)  It  does  not  seem  unreasonable  to  expect  a  vendor  to 
dedicate  his  system  for  up  to  eighty  hours  to  demonstrate  Its 
capabilities.  This  Is  particularly  valid  given  the  high  system 
acquisition  costs  DMA  must  accept  from  such  vendors.  The  other  side  of 
this  "problem"  Is  that  a  vendor  has  the  right  to  refuse  all  or  portions  of 
the  benchmark  package.  DMA  Mill  judge  a  system  not  only  on  benchmark 
testing  but  many  other  factors  as  Mell. 

Overall,  Battelle  feels  that  the  benchmark  testing  materials 
and  procedures  Implemented  at  DMAHTC  on  the  Scltex  Response-250  and 
Broomall  AGDS  Mere  effective.  Consideration  should  be  given  to  the 
suggested  Improvements.  HoMever,  DMA  noM  possesses  a  capability  to  assess 
the  performance  levels  of  state-of-the-art  cartographic  data  capture 
systems.  Additionally,  a  benchmark  for  performance  level  has  been 
established  for  the  Scltex  (and  AGDS)  at  DMAHTC.  Benchmark  test 
results  of  neM  ("and  Improved")  Scltex  and  other  commercial  systems  can  be 
compared  to  this  standard. 
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8.0  RECOMMENDATIONS  FOR  BENCHMARK  TESTING  OF  STATE -OF -THE-  ART 
CARTOGRAPHIC  DATA  CAPTURE  SYSTEMS 

There  exists  In  today's  marketplace  a  number  of  cartographic 
data  capture  systems  which  claim  state-of-the-art  capabilities.  The  DMA 
Raster- to-Vector  Analysis  final  technical  report  provides  an  overview  of 
these  systems.  It  describes  their  basic  system  facilities,  strengths  and 
weaknesses  based  on  commercial  literature  and  conversations  with 
"knowledgeable"  people  (both  vendors  and  users  alike).  These  observations 
are  not  based  on  empirical  facts. 

Battelle  believes  that  certain  commercially  available  systems 
are  worthy  candidates  for  benchmark  testing.  These  Include:  Scltex 
Response-280,  Intergraph  Scan  Data  Capture  System,  SYSSCAN  Kartoscan  and 
Laserscan  Lasertrak.  The  Scltex  Response-280  series  deserves 

benchmarking  to  ascertain  If  performance  enhancements  have  been  built  Into 
this  upgrade.  DMA's  current  Investment  In  Scitex  is  high  and  growing 
(DMAAC  acquisition  of  a  Scltex  Response  280  is  a  recent  example). 
Decisions  to  continue  and  grow  with  Scitex  technology  should  be  based  on 
some  empirical  data,  derivable  from  the  DMA  benchmark.  Conversations  with 
key  Intergraph  system  development  engineers  and  past  experience  with 
Intergraph  products  makes  a  strong  case  for  a  benchmark  testing 
recommendation.  Intergraph's  purported  use  of  new  raster- to- vector 
algorithms  and  hardware  processors  makes  this  newly  Introduced  system 
appear  quite  competitive.  SYSSCAN  Kartoscan  represents  another  commercial 
vendor  who  apparently  Is  Investing  In  an  on-going  program  of  system 
development  and  Improvement.  Of  particular  Interest  is  their  work  In 
Europe  with  both  DTED  and  DFAD  data  types.  They  have  developed  a  number 
of  processing  capabilities  specifically  designed  for  these  data  types, 
which  are  obviously  relevant  to  DMA  requirements.  The  final  system 
recommended  for  benchmark  testing  Is  the  Laserscan  Lasertrak.  This 
represents  the  only  line  following  system  to  be  recommended.  Its 
processing  of  contour  and  DFAD  data  should  he  tested  for  possible 
application  at  DMA.  Applications  development  at  USGS  on  their  newly 
acquired  Lasertrak  systems  should  he  observed  for  Indications  of  future 
DMA  applicability. 


9.0  RECOMMENDATIONS  FOR  FUTURE 
RESEARCH  AND  DEVELOPMENT 


Battel 1e  recommends  three  projects  for  future  research  and 
development.  The  first  recommendation  Is  to  support  the  completion  of  the 
quallty/edltlng  assessment  component  of  the  basic  raster-to-vector 
conversion  benchmark  testing  package.  The  second  recommendation  Is  to 
support  the  development  of  an  advanced  raster-to-vector  conversion 
benchmark  testing  package.  The  third  recommendation  Is  to  establish  a 
program  of  In-house  commercial  system  evaluation  and  optimization. 

9.1  Completion  of  the  Qualitative/Editing  Assessment 
Component  of  the  DMA  Raster-to-Vector 
Conversion  Benchmark  Testing  Package 

i 

The  qualitative/editing  assessment  component  is  a  unique  and 
pertinent  contribution  to  the  DMA  rister-to-vector  conversion  benchmark 
testing  capability.  This  Individual  test  sheet,  consisting  of  a  series  of 
geometric  patterns  of  "perfect"  and  degraded  form,  provides  a  mechanism 
for  testing  the  automated  editing  functions  of  state-of-the-art 
cartographic  data  capture  systems.  It  also  provides  a  means  to  focus  on 
the  qualitative  aspects  of  converting  analog  cartographic  features  Into 
digital  vector  data. 

Battel le  recommends  that  this  synthetic  cartographic  geometry 
test  sheet  which  Is  currently  partially  created,  be  completed  and 
validated  on  the  Scltex  and  AGDS  systems  at  DMAHTC.  Together  with  the 
existing  set  of  sixteen  benchmark  materials,  the  synthetic  cartographic 
geometry  test  sheet  will  provide  DMA  with  a  comprehensive  benchmark 
testing  capability. 
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9.2  Advanced  Raster-to-Vector  Conversion  Benchmark 
Testing  Package  Development* 

The  benchmark  testing  package  developed  by  Battel le  (including 
the  proposed  synthetic  cartographic  geometry  sheet)  represents  a 
comprehensive  testing  capability  for  basic  cartographic  input  possessing 
limited  symbology  or  geometric  complexity.  The  full  range  of  analog 
cartographic  geometries  and  symbolization  (e.g.,  cased  roads,  depression 
contours,  point  symbols,  dash-dot  patterns  for  political  boundaries,  swamp 
symbols,  railroad  ticks  and  tint  screens)  have  yet  to  be  addressed. 
Benchmark  materials  and  tests  will  be  required  to  evaluate  the  performance 
levels  of  automatic  feature  recognition  and  tagging  capabilities  in 
addition  to  spatial/topological  encoding  routines  being  touted  by 
state-of-the  art  cartographic  data  capture  system. 

Battel le  recommends  that  DMA  support  the  development  of  an 
advanced  cartographic  benchmark  testing  capability  integrated  with  the 
basic  package  already  provided  to  the  mapping  agency.  The  significant 
technological  advances  being  made  by  cartographic  state-of-the-art  data 
capture  systems  warrant  these  new  testing  mechanisms. 

9.3  Vendor  System  Evaluation  and  Optimization** 


Battel le  recommends  an  on-going  program  of  DMA  cartographic 
data  capture  system  (i.e.,  Scitex  and  AGDS)  evaluation  with  the  goal  of 
fully  optimizing  existing  facilities  and  specifying  areas  in  need  of 
further  development.  The  clear  definition  of  the  cartographic  utility  of 
all  available  software  routines  (in  particular  on  the  Scitex)  is  of 
primary  concern.  A  systematic  program  of  software  analysis,  cataloguing 
and  use  optimization  is  recommended.  A  closer  look  at  the  optimal  roles 


This  Is  a  restatement  of  a  recommended  future 
research  and  development  option  presented  in 
Section  8.1  of  the  DMA  Raster-to-Vector 
Analysis'  final  technical  report. 

This  is  a  restatement  of  a  recommended  future 
research  and  development  option  presented  in 
section  8.3  of  the  DMA  Raster-to-Vector 
Analysis'  final  technical  report. 


of  interactive,  computer-assisted  and  automatic  functions  Is  recommended. 
The  effective  use  of  batch  programming  and  heretofore  unused  processing 
functions  also  requires  further  Investigation.  The  generalized  functions 
of  all  commercial  data  capture  systems  (such  as  the  Scltex)  require  continued 
tailoring  to  the  specific  DMA  ana log- to- vector  conversion  requirements. 


